The present invention relates to a moving image combining apparatus combining computer graphics with video images.
The following processing is conventionally performed when combining and displaying computer graphics and video images in a virtual space displayed by a computer using three-dimensional (hereafter 3D) graphics. A computer graphics (hereafter CG) image is generated by performing graphics-generating calculations using coordinate values showing locations and outlines for objects in a virtual 3D space. A video frame is extracted from a video sequence, and pasted onto the generated CG image using a method known as texture mapping. The resulting image is then displayed. High-speed repetition of this processing sequence for generating of a CG image, extracting of a video frame, pasting the video frame onto the CG image and displaying the resulting image enables CG images on which video frames have been pasted to be displayed sequentially, giving the appearance of a moving image.
However, the respective display rates for computer graphics and video images prior to combining are not necessarily identical. In a video sequence, a fixed number of frames can be displayed during a fixed time (this is hereafter to as the display rate). One standard for the video image display rate is 30 frames per second. In contrast, for computer graphics, the calculation time required to generate a CG image from coordinate values for object locations and outlines varies according to the number of objects to be displayed. As a result, achieving a uniform display rate is normally difficult.
Suppose the video image and computer graphics display rates are respectively 30 and 10 frames per second and moving images are combined at the computer graphics display rate. This means that, of the 30 video frames that can potentially be displayed in one second, only the 10 frames coinciding with display times of CG images can be displayed. Consequently, the remaining 20 frames cannot be displayed, so that the movement of the video sequence is jerky.
If moving images are combined at the video image display rate, however, the calculation required to generate a CG image cannot be completed in the interval between the display of consecutive video frames, meaning that it may not be possible to generate a CG image on every occasion.
In order to overcome the above problems, an object of the present invention is to provide a moving image combining apparatus combining computer graphics and video images at their respective display rates, a moving image combining method, and a recording medium recording a program for combining moving images.
An invention achieving the above object is a moving image combining apparatus combining computer graphics images (hereafter referred to as CG images) and at least one video sequence composed of a plurality of video frames, the moving image combining apparatus including the following. An information storage unit storing object information showing an outline and location for at least one object in three-dimensional (3D) space, a video obtaining unit obtaining from an external source at least one video sequence composed of a plurality of video frames generated at a fixed video display rate, an image storage unit, a receiving unit for receiving position information showing a position of a moving viewpoint, a graphics generating unit for generating CG images one at a time at a graphics display rate and, on completing the generation of a CG image, writing the CG image into the image storage unit, the CG image obtained by projecting each object whose outline and location is shown by the object information onto a projection surface, as seen from a current position of the moving viewpoint shown by the position information, and a video frame generating unit for fetching at least one video frame from the at least one video sequence at the video display rate and writing the fetched at least one video frame over a CG image, the CG image being stored in the image storage unit immediately prior to the time that the at least one video frame was fetched.
This construction enables generating of CG images and decoding of video frames to be performed in parallel using separate processes, and the generated CG image and still video to be combined in the storage unit. As a result, computer graphics and video images can be combined at their respective display rates.
The graphics generating unit may further perform rendering on each generated CG image, and write the rendered CG images into the image storage unit.
This construction enables rendering to be performed on a CG image, so that realistic graphics can be obtained.
Here, the following construction may also be used. Each object includes at least one video display area. The moving image combining apparatus combines, on at least one video screen located on the projection surface, at least one video sequence and a CG image, each video screen corresponding to a video display area. The object information includes information showing an outline and location for each video display area. The graphics generating unit further calculates screen information showing an outline and location for each video screen, each video screen obtained by projecting a video display area shown by an outline and location in the object information onto the projection surface. The video frame generating unit overwrites fetched video frames at each location shown by the screen information, so that each fetched video frame fits an outline shown in the screen information.
This construction enables a video sequence to be combined on a video screen of an object.
The video frame generating unit may also be constructed so that it generates transformed video frames by transforming the fetched video frames to fit an outline shown in the screen information; and overwrites the transformed video frames into the image storage unit.
This construction enables a video frame to be transformed to fit the video screen of the object, enabling the video sequence to be combined more realistically.
The following construction may also be used. Each object has a plurality of video display areas. The video obtaining unit obtains a plurality of video sequences from an external source. The moving image combining apparatus combines, on each of a plurality of video screens on a projection surface, one of the video sequences with a CG image, each video screen corresponding to one of the plurality of video display areas. The object information includes information showing outlines and locations for a plurality of video display areas. The graphics generating unit calculates screen information for each piece of information showing the outline and location for one of the plurality of video display areas. The video frame generating unit fetches video frames from each of the plurality of video sequences, and overwrites fetched video frames from the different video sequences at the different locations shown by the plurality of pieces of screen information, so that the fetched video frames fit the outlines shown in the screen information.
This construction enables video sequences to be combined on each of a plurality of video screens, when an object has a plurality of video display areas.
The video frame generating unit may also include the following. A priority ranking determining unit for determining a priority ranking for each video screen based on the plurality of pieces of calculated screen information. A video decoding unit for obtaining video frames from each of the plurality of video sequences, based on the determined priority ranking. A masking location calculating unit for calculating locations to be masked on each video screen, based on the plurality of pieces of calculated screen information and the priority ranking determined for each video screen. A masking unit for masking the transformed video frames at the calculated locations. Here, the video frame generating unit overwrites the transformed video frames which have been masked into the image storage unit.
This construction enables priority rankings to be determined according to video screens of objects, video frames to be obtained from video sequences based on the priority rankings, and masking to be performed on each video screen, so that video sequences can be combined more realistically.
The priority ranking determining unit may determine priority rankings using the plurality of pieces of calculated screen information, with video screens nearer to the viewpoint having a higher priority ranking.
This construction enables video screens nearer to a viewpoint to be given a higher priority ranking, so that video sequences can be combined more realistically.
The priority ranking determining unit may determine priority rankings using the plurality of pieces of calculated screen information, with video screens calculated as having a larger surface area having a higher priority ranking.
This construction enables video screens with a larger area to be given a higher priority ranking, so that a higher quality picture can be obtained.
The video decoding unit may obtain all of the video frames from a video sequence with the highest priority ranking, and omit more video frames from video sequences with lower priority rankings.
This construction enables a greater number of frames to be skipped at lower priority rankings, so that the picture quality of decoded video frames can be adjusted according to the priority ranking.
The video decoding unit may include an image quality adjustment unit reducing luminance of obtained video frames, and does not reduce the luminance of video frames from the video sequence with the highest priority ranking, while reducing the luminance of video frames from video sequences with lower priority rankings.
This construction enables luminance to be decreased at lower priority rankings, so that flickering is not noticeable for lower-ranked video display surfaces likely to have a low display rate.
The invention may also be a moving image combining apparatus combining three-dimensional CG images and at least one video sequence composed of a plurality of video frames. The moving image combining apparatus includes the following. An information storage unit for storing object information showing an outline and location for each object, and an outline and location for at least one video display area for each object, a video obtaining unit for obtaining from an external source at least one video sequence composed of a plurality of video frames generated at a fixed video display rate, a CG image storage unit, a video frame storage unit, an image storage unit, a receiving unit for receiving position information showing a position of a moving viewpoint, a graphics generating unit for generating CG images one at a time at a graphics display rate and on completing the generation of a CG image, writing the CG image into the CG image storage unit, the CG image obtained by projecting each object whose outline and location is shown by the object information onto a projection surface, as seen from a current position of the moving viewpoint shown by the position information; and calculating screen information showing an outline and location for at least one video screen obtained by projecting each video display area shown by an outline and location in the object information onto the projection surface, a video frame generating unit for fetching at least one video frame from the at least one video sequence at the video display rate and overwriting the fetched at least one video frame into the video frame storage unit, and a selecting unit for selecting elements forming still images from the at least one video frame written in the video frame storage unit and a CG image written in the CG image storage unit, the CG image being written in the CG image storage unit immediately prior to the time that the at least one video frame was fetched; and writing the selected elements in the image storage unit.
This construction enables generating of CG images and decoding of video frames to be performed in parallel as separate processes. As a result, computer graphics and video images can be combined at their respective display rates, and images can be combined via a selection signal, so that the construction of the apparatus can be simplified.
The CG image storage unit may include a first graphics storage unit and a second graphics storage unit and the video frame storage unit may include a first video storage unit and a second video storage unit. The graphics generating unit writes obtained CG images alternately in the first and second graphics storage units. The video frame generating unit writes obtained video frames alternately in the first and second graphics storage units. The selecting unit reads a CG image from the second graphics storage unit while the graphics generating unit is writing a CG image into the first graphics storage unit, and reads a CG image from the first graphics storage unit while the graphics generating unit is writing a CG image into the second graphics storage unit. The selecting unit also reads a video frame from the second video storage unit while the video frame generating unit is writing a video frame into the first video storage unit, and reads a video frame from the first video storage unit while the video frame generating unit is writing a video frame into the second video storage unit. Then the selecting unit selects elements forming still images from the read CG images and video frames.
This construction enables generating of CG images, decoding of video frames, and combining of generated CG images with video frames to be performed in parallel as separate processes. As a result, computer graphics and video images can be combined at their respective display rates, and generating of CG images, decoding of video frames and combining of CG images with video frames can be performed more quickly.
The graphics generating unit may further perform rendering on each generated CG image, and write the rendered CG images into the image storage unit.
This construction enables graphics to be rendered, so that computer graphics can be generated more realistically.
The video frame generating unit may generate transformed video frames by transforming the fetched video frames to fit an outline shown in the screen information; and overwrite the transformed video frames into the image storage unit.
This construction enables video frames to be transformed to fit the outline of the video screen of each object, so that a video sequence can be combined more realistically.
The following construction may also be used. Each object has a plurality of video display areas. The video obtaining unit obtains a plurality of video sequences from an external source. The moving image combining apparatus combines, on each of a plurality of video screens on a projection surface, one of the video sequences with a CG image, each video screen corresponding to one of the plurality of video display areas. The object information includes information showing outlines and locations for a plurality of video display areas. The graphics generating unit calculates screen information for each piece of information showing the outline and location for one of the plurality of video display areas. The video frame generating unit fetches video frames from each of the plurality of video sequences, and overwrites fetched video frames from the different video sequences at the different locations shown by the plurality of pieces of screen information, so that the fetched video frames fit the outlines shown in the screen information.
This construction enables video images to be combined on a plurality of video screens of objects, when an object has a plurality of video display areas.
The video frame generating unit may include the following. A priority ranking determining unit for determining a priority ranking for each video screen based on the plurality of pieces of calculated screen information. A video decoding unit for obtaining video frames from each of the plurality of video sequences, based on the determined priority ranking. A masking location calculating unit for calculating locations to be masked on each video screen, based on the plurality of pieces of calculated screen information and the priority ranking determined for each video screen. A masking unit for masking the transformed video frames at the calculated locations. Here, the video frame generating unit overwrites the transformed video frames which have been masked into the image storage unit.
This construction enables priority rankings to be determined according to video screens of objects, video frames to be obtained from video sequences based on the priority rankings, and masking to be performed on each video screen, so that video sequences can be combined more realistically.
The priority ranking determining unit may determine priority rankings using the plurality of pieces of calculated screen information, with video screens nearer to the viewpoint having a higher priority ranking.
This construction enables video screens nearer the viewpoint to be given a higher priority ranking, so that video sequences can be combined more realistically.
The priority ranking determining unit may determine priority rankings using the plurality of pieces of calculated screen information, with video screens calculated as having a larger surface area having a higher priority ranking.
This construction enables video screens with larger areas to receive a higher priority ranking, so that picture quality can be increased.
The video decoding unit may obtain all of the video frames from a video sequence with the highest priority ranking, and omit more video frames from video sequences with lower priority rankings.
This construction enables a larger number of frames to be skipped at lower priority rankings, so that picture quality for decoded video frames can be adjusted according to the priority ranking.
The video decoding unit may include an image quality adjustment unit reducing luminance of obtained video frames, and does not reduce the luminance of video frames from the video sequence with the highest priority ranking, while reducing the luminance of video frames from video sequences with lower priority rankings.
This construction enables luminance to be decreased at lower priority rankings, so that flicker is not noticeable for lower-ranked video display surfaces likely to have a low display rate.
A moving image combining method for combining CG images and at least one video sequence composed of a plurality of video frames may also be used. The moving image combining method is used by a moving image combining apparatus having an information storage unit and an image storage unit, the information storage unit storing object information showing an outline and location for at least one object in three-dimensional space. The moving image combining method includes the following. A video obtaining step obtaining from an external source at least one video sequence composed of a plurality of video frames generated at a fixed video display rate. A receiving step receiving position information showing a position of a moving viewpoint. A graphics generating step generating CG images one at a time at a graphics display rate and, on completing the generation of a CG image, writing the CG image into the image storage unit, the CG image obtained by projecting each object whose outline and location is shown by the object information onto a projection surface, as seen from a current position of the moving viewpoint shown by the position information. A video frame generating step fetching at least one video frame from the at least one video sequence at the video display rate and writing the fetched at least one video frame over a CG image, the CG image being stored in the image storage unit immediately prior to the time that the at least one video frame was fetched.
When using this method, the same effects are apparent as for the moving image combining apparatus.
A moving image combining method for combining, on an video display area, CG images and at least one video sequence composed of a plurality of video frames may also be used. The moving image combining method is used by a moving image combining apparatus having an information storage unit, a CG image storage unit, a video frame storage unit, and an image storage unit, the information storage unit storing object information showing an outline and location for at least one object, and an outline and location for a video screen for each object, in three-dimensional space. The moving image combining method includes the following. A video obtaining step obtaining from an external source at least one video sequence composed of a plurality of video frames generated at a fixed video display rate. A receiving step receiving position information showing a position of a moving viewpoint. A graphics generating step generating CG images one at a time at a graphics display rate and, on completing the generation of a CG image, writing the CG image into the graphics storage unit, the CG image obtained by projecting each object whose outline and location is shown by the object information onto a projection surface, as seen from a current position of the moving viewpoint shown by the position information and calculating screen information showing an outline and location for at least one video screen, the video screen obtained by projecting the at least one video display area shown by an outline and location in the object information onto the projection surface. A video frame generating step fetching at least one video frame from the at least one video sequence at the video display rate and overwriting the fetched at least one video frame in the video frame storage unit. A selecting step selecting elements forming still images from the at least one video frame written in the video frame storage unit and a CG image written in the CG image storage unit, the CG image being written in the CG image storage unit immediately prior to the time that the at least one video frame was fetched, and writing the selected elements in the image storage unit.
When using this method, the same effects are apparent as for the moving image combining apparatus.
The invention may also be realized using a recording medium recording a moving image combining program combining CG images and at least one video sequence composed of a plurality of video frames. The moving image combining program used by a computer having an information storage unit and an image storage unit, the information storage unit storing object information showing an outline and location for at least one object in three-dimensional space. The moving image combining program including the following. A video obtaining step obtaining from an external source at least one video sequence composed of a plurality of video frames generated at a fixed video display rate. A receiving step receiving position information showing a position of a moving viewpoint. A graphics generating step generating CG images one at a time at a graphics display rate and, on completing the generation of a CG image, writing the CG image into the image storage unit, the CG image obtained by projecting each object whose outline and location is shown by the object information onto a projection surface, as seen from a current position of the moving viewpoint shown by the position information. A video frame generating step fetching at least one video frame from the at least one video sequence at the video display rate and writing the fetched at least one video frame over a CG image, the CG image being stored in the image storage unit immediately prior to the time that the at least one video frame was fetched.
When this program is executed by a computer, the same effects are apparent as for the moving image combining apparatus.
The invention may also use a recording medium recording a moving image combining program combining, on a video display area, CG images and at least one video sequence composed of a plurality of video frames. The moving image combining program used by a computer having an information storage unit, a CG image storage unit, a video frame storage unit, and an image storage unit, the information storage unit storing object information showing an outline and location for at least one object, and an outline and location for a video display area for each object, in three-dimensional space. The moving image combining program includes the following. A video obtaining step obtaining from an external source at least one video sequence composed of a plurality of video frames generated at a fixed video display rate. A receiving step receiving position information showing a position of a moving viewpoint. A graphics generating step generating CG images one at a time at a graphics display rate and, on completing the generation of a CG image, writing the CG image into the graphics storage unit, the CG image obtained by projecting each object whose outline and location is shown by the object information onto a projection surface, as seen from a current position of the moving viewpoint shown by the position information and calculating screen information showing an outline and location for at least one video screen, the video screen obtained by projecting the at least one video display area shown by an outline and location in the object information onto the projection surface. A video frame generating step fetching at least one video frame from the at least one video sequence at the video display rate and overwriting the fetched at least one video frame in the video frame storage unit. A selecting step selecting elements forming still images from the at least one video frame written in the video frame storage unit and a CG image written in the CG image storage unit, the CG image being written in the CG image storage unit immediately prior to the time that the at least one video frame was fetched, and writing the selected elements in the image storage unit.
When this program is executed by a computer, the same effects are apparent as for the moving image combining apparatus.